A Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks

Why heavily parameterized neural networks (NNs) do not overfit the data is an important long standing open question. We propose a phenomenological model of NN training to explain this non-overfitting puzzle. Our linear frequency principle (LFP) accounts for key dynamical feature NNs: they learn low frequencies first, irrespective microscopic details. Theory based on our LFP shows that dominance target functions condition NNs and verified by experiments. Furthermore, through ideal two-layer NN, we unravel how detailed dynamics statistically gives rise with quantitative prediction power.